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SPECIFICATION 

Electronic Version 1 .2.8 
Stylesheet Version 1 .0 

Method and Apparatus of 
Universal Remote Pointing 
Control for Home Entertainment 
System and Computer 

Background of Invention 

[0001] 

With advancing technology, more and more features are added to home video and 
audio entertainment systems. For example, interactive television sets allow users to 
purchase a pay program by pressing buttons on the remote control. However, the rich 
set of functions requires more buttons on the remote control unit. The jam-packed 
button layout on the remote control unit makes the handheld device bulky and 
complicated. Moreover, an increasing number of audio and video component devices, 
for example, VCRs, DVD players, digital TV set-top boxes, are added into home 
entertainment systems. Each device is usually controlled by a unique remote control 
unit. To reduce the user's confusion of multiple remote control units, universal remote 
control devices were introduced to consumers. The universal remote control device 
can be either preprogrammed or trained with other remote controls by the user to 
provide multi-device control functionality. However, because more functions are being 
added to this type of handheld device, and because of the limited number of buttons 
available (which are already crowding the device), each button must serve multiple 
functions. Unfortunately, the multi-function buttons cannot provide clear visual 
feedback indicating their current function. This unfriendly user interface is obscure to 
the user operating the remote control unit and leads to only a small subset of the 
functions being utilized. Furthermore, the expandability of present universal remote 
control devices is very poor. As new media modules are introduced into home 
entertainment systems, for instance, Internet browsers, it becomes even more difficult 



APP ID-1 0065798 



Page 1 of 26 



;li r:i: ft 53 7' ^Qi e . ± .1. ii^IJO E? 



to adapt the existing universal remote control to the new requirements, in the case of 
Internet browsers, that users be able to move a pointer and select a visual object on 
the screen to operate a certain function. A handheld pointing control device is 
desirable In such a case. While using the pointing device, the on-screen graphical user 
interface (GUI) provides friendly visual feedback. The dynamically displayed selectable 
on-screen identifiers (menus, icons, buttons, etc.) greatly reduce the number of 
buttons on the pointing control device. 

[0002] In the case of computer slide presentations, a convenient handheld remote 

pointing and control device is also considered necessary. Conventional computer 
control depends on keyboard and mouse which are physically bounded with computer 
hardware and a fixed surface such as a table. To control the flow of the presentation 
slides or to point out figures on the slide to the audience, the presenter is forced to 
stay with the computer keyboard and mouse. This constraint is very inconvenient for 
the presenter trying to deliver his/her talk to the audience. A remote pointing control 
device could help the presenter to freely walk about the stage and move a pointer on 
the screen to guide the audience. 

[0003] Because of the need for a remote pointing mechanism for home entertainment 

systems and computer presentations, many methods and devices have been invented. 
For examples, Fan (U.S. Patent 5,926,168) has described several methods, including 
using light emission and electromagnetic fields, to develop remote pointing devices; 
Kahn (US Patent 6,404.41 6) described a pointing interface for computer systems 
based on raster scanned light emitted from display screens. The methods presented in 
those inventions are complicated, and some require a new display apparatus to 
replace the existing one. Marsh et al. (U.S. Patent 5,999,1 67) introduced a pointer 
control device based on an ultrasound source. Pilcher et al. (U.S. Patent 5,3 59,348), 
Hansen (U.S. Patent 5,045,843), Odell (U.S. Patent 5.574,479) and King, et al (U.S. 
Patent 4,565,999) presented pointing devices based on detecting fixed light sources. 
Auerbach (U.S. Patent 4,796,01 9) explained a pointing device containing multiple light 
sources and the lights are detected by a fixed light sensor. Wang et al. (U.S. Patent 
5.126,513) suggested a pointing measurement method by detecting the wave phases 
from a fixed transmitter. However, in practice, all the approaches based on detecting 
fixed local sources suffer from the limitations of the fixed source locations and 
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orientations, as well as the distance between the pointing device and fixed sources. 
Moreover, the control methods proposed in all the aforementioned inventions are 
limited to only a single target device. The control scope is narrow and cannot cover all 
the related video/audio devices or equipment. 

[0004] Recently, low cost magnetic field sensors based on magneto-resistive, magneto- 
inductive and Hall-effect technologies were developed. Those magnetic sensors are 
sensitive enough to measure earth's magnetic field and are widely used in such 
navigational devices as digital compasses and the Global Positioning System (CPS). 
Some magnetic sensors are packaged to detect two-axis, even three-axis, magnetic 
field changes and provide a linear output to the direction of the magnetic field flux, 
such as HMC1052 two-axis magnetic sensor from Honeywell 

(www.ssec.honeywell.com). The two-axis magnetic field sensor can be easy and cost- 
perfect to implement a pointing device to detect the yaw (azimuth) angle relative to 
earth's North Pole. However, using magnetic field sensors to detect a pitch 
(inclination) angle change would be a problem, particularly when the pointing device's 
heading direction is perpendicular to earth's North-South axis. Hall et al. (U.S. Patent 
5,703,623) presented a pointing device using three pairs of orthogonally mounted 
one-axis Hall-effect sensors. To overcome the problem in measuring pitch and roll 
ahglies, a set of piezoelectric sensors is used to detect the acceleration changes. The 
authors suggested using the detected acceleration data to compensate the deficient of 
magnetic sensors. However, to measure device angular movement an integration of 
the acceleration steps is required. The piezoelectric sensors detect only the dynamic 
changes of acceleration. The acceleration measurement errors are introduced because 
piezoelectric sensors are failed to measure the constant acceleration. The . 
accumulated acceleration error in the integration process would eventually render the 
device unusable. 

[0005] detect a pointing device's pitch and roll angles, a static accelerometer can be 

used. Recently, low-cost, lightweight accelerometer sensors using Micro- Electro- 
Mechanical Systems (MEMS) technology are available from many sources. MEMS 
devices integrate mechanical elements, sensors, actuators, and electronics on a 
common silicon substrate using micro-fabrication technology, which provides a cost- 
effective and small-footprint component for consumer manufactories. Two-axis linear 
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MEMS accelerometers. such as ADXL-202E from Analog Devices (www.analog.com), 
LIS2L01 from STMicroelectronics (www.st.com). and MXD201 OU/W from MEMSIC 
(www.memsic.com). can measure both dynamic and static acceleration and are good 
candidates for use in pointing devices to determine the pitch and roll angles. The 
earth's gravity exerts a constant acceleration on the MEMS accelerometer. By 
calculating the accelerometer's static acceleration outputs, a tilt angle (pitch or roll) 
can be obtained. 

[0006] Besides magnetic field sensors and accelerometer sensors, gyro sensors can also 
be used in pointing device design. Gyro sensors, such as the ADXRSl 50 MEMS 
gyroscope from Analog Devices (www.analog.com), can detect changes in the device's 
orientation angle and thus can be used in detecting the pointing device's heading. 

[0007] The object of the present invention is to provide a low-cost, practical, universal 
pointing device to control home entertainment systems and computer systems using 
spatial orientation sensor technologies. 

Summary of Invention 

[0008] ^ universal pointing control system for televisions and computer displays is 

disclosed. The system is comprised of a remote handheld device, a display control 
unit and a command delivery unit. The remote handheld device includes a set of 
orientation sensors that detect the device's current orientation. In the preferred 
embodiment, a two-axis magnetic sensor identifies the device's azimuth angle by 
detecting the earth's magnetic field, and a dual-axis accelerometer sensor identifies 
the device's inclination angle by detecting the earth's gravity. The signals from the 
orientation sensors are translated and encoded into pointing direction information by 
a microprocessor or logic circuits on the pointing device and transmitted to the 
display control unit. Along with the directional information, data regarding the user's 
selection activities collected by a selection unit in the handheld device is also encoded 
and sent to the display unit. The display control unit includes a data transceiver, a 
CPU, and a display control circuit for interfacing target device. The pointing direction 
information received by the transceiver is decoded and manipulated by the on board 
CPU. Based on the pointing information, the CPU instructs the controlled target device 
interface, either a television set or a computer, to display a pointer at the 
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corresponding coordinates on the target device screen. User selection activities are 
also interpreted by the CPU based on the current pointer location, and corresponding 
commands are sent to the command delivery unit. The command delivery unit, which 
can be a stand-alone device or built into the handheld pointing device, forwards the 
commands to any remote controllable target device using an infrared beam to execute 
a desired operation. 

[0009] The handheld remote control device is simple and easy to use. User directly points 
to any position of the screen and a cursor is displayed on the screen at the pointed 
location. By selecting a menu or active control shape on the screen using a selection 
button on the device, the user can control the target device's operation intuitively. 
Because fewer buttons are required to operate the device (e.g. a selection button, a 
calibration button, and a button to show and hide the on-screen pointer), the device 
can be made smaller and lighter. The selectable items can vary and change their 
appearance dynamically based on the status of the operations. With visual feedback, 
the system provides a much better and friendlier graphical interface to users. Because 
the pointing signals are generated from the handheld remote control device without 
reference to any source from other devices or equipment, there is no significant 
change necessary on the television or computer system. In the described embodiment, 
the remote pointing device can be directly used in existing televisions and computers 
without any modification. The control scope of this system is broad enough to cover 
all the audio/video devices which are originally controlled by their respective remote 
controls. The extendibility of the system allows new types of devices to be easily 
adapted and controlled. 



Brief Description of Drawings 



[0010] 



FIG. / Is a perspective view of the universal pointing system in controlling a variety 
of equipments in the home entertainment system. 



[001 1] 



FIG. 2 is a perspective view of the pointing system in controlling computer 



presentations. 



[0012] 



FIG. 3 shows the components in the handheld pointing device. 



[0013] 



FIG. 4 a and ^Z? demonstrate the principal mechanism of the orientation sensor 
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detecting the device's orientation changes, and how the screen pointer to reflects 
these changes. 

[0014] FIG. 5 is the functional block diagram of the pointing device. 

[001 5] FIG. 5 a is the functional block diagram of the display control unit for a computer. 

[0016] FIG. 6 b \s the functional block diagram of the display control unit for a home 
entertainment system. 

[0017] FIG. 7a is the functional block diagram of the command delivery unit. 

[001 8] FIG. 7b shows the command delivery unit being trained by an original remote 
control. 

[0019] FIG. Sa is the alternative functional block diagram of the display control unit which 
includes the remote control training circuit. 

[0020] FIG. Sb shows the display control unit being trained by an original remote control. 

Detailed Description 

[0021] 

The present invention's universal pointing control system consists of a handheld 
pointing device 7 00, a display control unit 200 and a command deliver unit 300 as 
shown in F/G. 1 . In this example, the display control unit 200 \s connected to a 
television 400 and a video component device SOO , which can be a digital TV set-top 
box, a VCR, or a DVD player, through video cables S20 and 510 , respectively. The 
display control unit 200 can also be embedded inside the TV or other video 
component device in alternative embodiments. The handheld pointing device WO \s 
aimed at the television screen 420 indicated by a line of sight 10 . On the other end of 
this line, a pointer 410 \s displayed on the screen. When the user points the device to 
an arbitrary position of the screen, a set of orientation sensors inside the pointing 
device 100 , which will be described in later, detects the device's current orientation 
and generate the pointing direction signal. The pointing direction signal is encoded 
and sent to the display control unit 2<7<7through a transmission link 5^?. This 
transmission link can be any form of signal linkage. For example, it could be 
implemented by using radio frequency (RF) wireless link, infrared (IR) link, or even a 
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wired cable. Upon receiving tlie signal, a central process unit (CPU) inside the display 
unit 200 decodes and analyzes the pointing direction and determines the new 
coordinates of the pointer on the screen. A pointer is drawn at the calculated 
coordinate and the pointer image is then superimposed onto the input video signal, 
which is input from video component device SOOthrough cable SIO . A set of menus 
and control items 430 are also drawn and superimposed to the video signal. The 
composite video is then output to the television ^^7£7 through the output video cable 
S20 and displayed on the television screen 420 . As a result, the pointer 4 JO is shown 
at a new location on the screen where the user points to. The user perceives that the 
pointer is moved following the aiming line of sight 10 , 

[0022] Buttons are located on the handheld pointing device to collect the user's selection 
activities. Three buttons are shown in this example, one for command selection 
( 1 01 ) , one to show and hide screen pointer ( 102 ), and another one for calibration 
purpose ( 103). When the user uses the device at first time, a calibration procedure is 
performed. The user aims the device at the center of the screen and presses button 
103 . The device's pointing direction information is recorded and stored into the 
display control unit as the screen center reference. Any subsequent pointing 
information is then compared with this reference, and the difference will be calculated 
as the pointer displacement distance away from the screen center. 

[0023] During normal usage, as the user points and clicks the selection button, the on- 
screen menu or selectable items under the pointer are processed by the CPU in the 
display control unit. Selection information is generated and forwarded to the 
command delivery unit 300h\/ means of transmission link 5^?.The link 5(9, again, can 
be any form of signal linkage. The command deliver unit 300 can be a stand-alone 
device facing the TV 400 awd other equipments ( 500,510,520), or can be embedded 
inside the pointing device 100 . All remote control command codes for the devices in 
the home entertainment system are prerecorded in a memory module in the command 
delivery unit 300 . Upon receiving selection information, the command delivery unit 
issues the corresponding command by searching the memory module, and emits the 
command infrared (IR) signal through the IR emitter 351 to the controlled equipments. 
The target equipment performs a task as if it had received a command directly from 
its original remote control device. 
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[0024] F/C. 2 shows the pointing control system as used in a computer presentation 

scenario. In this case, the presentation is projected onto the screen 720 by a projector 
700 , which receives the video input from a computer 600 though a video cable 620. 
The display control unit 200 \s connected to the peripheral port of a computer 600 
through the cable 670 . The presenter aims the pointing device at the screen 720 by a 
line of sight 70 . The aiming direction information generated by a set of orientation 
sensors in the pointing device lOO ls transmitted to display control unit 200 through 
. transmission link SO . The CPU in the display control unit interprets the direction 
information, sends the pointer move command to the computer's peripheral port, and 
instructs the computer to move the pointer 710 on screen to the aimed place. This is 
analogous to moving the pointer by moving a regular computer mouse device, except 
that the moving information is in absolute coordinates instead of relative steps. The 
buttons 10/ , 702 , and 103 on the pointing device allow the presenter to select and 
execute a command remotely. 

[0025] F/C. J exposes the components inside of the handheld pointing device. On the top 
face of the device are buttons 701 , 702 , and 703 for collecting user selection 
activities. A set of orientation sensors /-?6?and 730 mounted on the print circuit board 
7 60 detect device's orientation changes. Note that the sensors are mounted 
orthogonally to each other. The sensor 72^7 detects the device's yaw (azimuth) angle 
and sensor 7 detects device's pitch (inclination) angle. Additional sensors (not show 
in the picture) could be used to detect device's roll angle which may provide an 
additional dimension of control. A microcontroller 7 7^7 provides computation power 
for calculating and encoding the orientation signal output from the orientation 
sensors. It also provides logic control for the transmitter 740 and other electronic 
components. The device is powered by batteries 770. 

[0026] orientation sensors' mechanisms are shown in F7G, 4a and 4b . The 

orientation sensor demonstrated in F7C. 4a is a magnetic field sensor, whereas the one 
in F/G. 4b \s an accelerometer sensor. However, the orientation detection may not be 
limited to these types of sensors. Other sensors, for example, a gyro sensor, can also 
be used in the pointing control system. In F7G. 4a , a two-axis magnetic field sensor 
720 is used to detect the device's orientation relative to the direction of the earth's 
magnetic field 25 . The sensor contains two magnetic field detectors which are 
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arranged orthogonal to each other. The sensor is mounted on the device's circuit 
board so that the two magnetic field detectors are laid on the x-z plane as shown in 
the picture. The azimuth angle c|> between the device's heading direction and the 
earth's North Pole direction can be calculated from the sensor's x and 2 output: <|> = 
arctan(x/z). When the user performs calibration, the device records the azimuth angle 
cb as the reference angle as the user points the device to center of the screen. When 
the device is rotated about the y-axis and the pointing direction Is moved away from 
screen's center, the azimuth angle difference from the reference angle Is <|) - <)> ^ 
This difference is Interpreted by the display control unit as the degree of the pointer's 
horizontal departure from the screen center. The amount by which the pointer moves 
horizontally ( 22 )can be adjusted in the display control unit proportionally to the 
change in the pointer's azimuth angle 2 J . 

[0027] The orientation sensor 130 uses a similar method to detect the device's inclination 
angle. The sensor could be an accelerometer or another orientation sensor that can 
sense the device's heading change in the y-z plane. An accelerometer sensor which 
can detect static acceleration Is described in detail here. The accelerometer sensor 

contains two orthogonally arranged acceleration detectors. The sensor is 
mounted perpendicular to the circuit board's plane so that one detector in the sensor 
detects y-axis acceleration and the other detects z-axis acceleration. Earth's gravity 
. 25 exerts a static acceleration on these detectors. When the device Is placed on a 
horizontal level, the accelerometer's z-axis detector outputs zero acceleration, while 
the y-axIs outputs the maximum acceleration (Ig). If the device Is rotated about the 
X-axis, the z and y channel outputs of the sensor are changed according to the 
inclination angle. The Inclination angle e thus can be calculated: e = arctan(z/y). 
During the calibration, the device's inclination angle to the screen center € ^ Is 
recorded and stored as a reference angle. Any Inclination angles sampled thereafter Is 
compared with this reference angle by determining the offset, ^ - e ^ . This difference 
is Interpreted by the display control unit as a degree of the pointer's departure from 
the screen's center in the vertical direction. The amount by which the pointer moves 
vertically ( 32) can be adjusted in the display control unit proportionally to the change 
in the pointer's inclination angle 3? . 

[0028] £^ simplified version, a one-axis accelerometer sensor can be used. In such a 
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case, the acceleration detector is mounted along the device's z-axis. The inclination 
angle e thus can be calculated: € = arcsin(z). 

[0029] F/G, 5 is the functional block diagram of the handheld pointing device. The signal 
conditioning circuit for sensor 72^7 consists of two amplifiers 12/ , / 23 and two low 
pass filters 122 , 124 . Because we are interested in the static position and low 
frequency movement of the device, the high frequency noises of the amplified x-axis 
and y-axis signals are filtered in order to get a higher resolution of the azimuth angle 
changes. Two amplifiers /J7 , /33 and two low pass filters 132 , 134 are for 
conditioning the sensor /Jt? output's x~axis and z-axis signals. We are interested in 
the sensor's static output relative to earth's constant gravity. Therefore, the high 
frequency noises of the signals are also filtered in order to get a higher resolution of 
the Inclination angle changes. The conditioned signals from sensors 72^ and 1 30 are 
then sent to an analog-to-digltal converter (ADC) 77 / by an analog multiplexer f 12 . 
The digitized sensors data are then sent to a microcontroller (MCU) 110 for further 
signal processing. Some variations of the orientation sensor convert the analog signal 
internally to a digital or time period-based signal. In those cases, the signals can be 
directly sampled by the microcontroller without an ADC chip. The MCU 7 7^7 computes 
the azimuth and inclination angles. Buttons 107 , 102 , and 103 produce activity 
signals that are sampled by MCU 100 , The sensor orientation data and buttons 
activities are coded in such a way that the display control unit can decode them later. 
The encoded data is passed to a modulator F 13 to modulate a carrier frequency for 
transmission. The transmitter 74^? emits the modulated signal SOlo the display 
control unit 200 . The circuit is powered by batteries 770 . A battery manager unit 777 
conditions the voltage for all components of the circuit. The MCU 7 7<7 constantly 
monitors the changes of sensor outputs. If the sensor outputs are not changed for a 
period, the MCU interprets the device as not been used. The MCU instructs the battery 
manager unit 777 to shut down the battery power supply to the transmitter and other 
components in order to save power consumption during the idle stage. 

[0030] show the functional block diagrams of display control unit 200 

for a computer and a television set, respectively. A central process unit (CPU) 270 , a 
receiver 227 , a demodulator 237 , and a memory module 270 are common for both 
cases. The transmitted signal SO from the pointing device, which includes handheld 
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device orientation and user selection activities, is intercepted by the receiver module 
227 . After being demodulated by a demodulator 237 , the pointing device data is 
sent to CPU 270 for further processing. The CPU compares the device's azimuth and 
inclination angle data with the reference angles, which are sampled and stored in the 
memory module 270 during the calibration procedure. The difference angles 
calculated are translated into screen coordinates and the target device is instructed to 
move the pointer to the new location. The interface components of the display control 
unit are different for each control target. In F7a 6a , a computer peripheral interface 
module is used to connect to a computer port. The pointer coordinates are sent to the 
computer and, by the computer's processor and video card, the pointer on the screen 
is moved to the corresponding location. The button activities are also sent to the i 
computer though this interface and trigger certain actions for the computer. 

[0031] F7C. demonstrates the display control Interfaces to a television. The input video 
signal, which may come from other home entertainment devices such as digital TV 
set-top boxes, DVD players, etc., is decoded by a video decoder 257 frame by frame. 
A new pointer image is drawn at the coordinate calculated by CPU 270 .Th^ pointer 
image, along with menus and other control item images pre-stored in the memory 
module 270 , are sent to a graphic-video multiplexer 250 to superimpose onto a 
video frame. The composite video frame is then encoded by a video encoder 252 and 
sent to the television for display. The process is about 30 frames per second. As the 
result, a pointer is moved on top of the video following the handheld device's pointing 
direction. If the CPU 2/6? senses a button click activity while the pointer is moved on 
top a menu or a controllable item, it sends a command to a transmitter 222 through a 
modulator 232 . The modulated transmission signal 60 \s forwarded to the command 
delivery unit 300 for controlling the television and other home entertainment 
equipment. 

[0032] fzfQ 73 is the functional block diagram of command delivery unit 300 . A receiver 
320 intercepts the transmitted signal from the display control unit 200 . The signal is 
sent to a microcontroller (MCU) after demodulation by a demodulator 330 . In the 
command delivery unit, there is a non-volatile memory module which stores all the 
control command codes for varieties of home entertainment equipments. These 
command codes can preset by the vendor or stored by the user during programming 
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or training procedures. The command codes are stored in such way that each 
command is coupled with an identification number (key). The arriving signal from the 
display control unit is served as the key so that MCU 3 10 can look up the key's record 
in memory and fetch the corresponding command code. For example, if the pointer is 
moved on top of a VCR play button on the screen and user clicks the selection button, 
the display control unit sends a value equal to 1 00 to the command delivery unit. By 
looking up the key value 1 00 in, the memory module, the MCU 370 fetches the pre- 
stored VCR play command code. The command code is sent to an infrared transmitter 
J5^7to drive an infrared emitter 3SI . The infrared -carried command code is then sent 
to the home entertainment equipment, in this case a VCR, to control their functions. 

[0033] F/C. 7/& demonstrates the programming (training) procedure of the command 
delivery unit. When the user adds new home entertainment equipment, he/#she can 
program the command delivery unit to learn commands for the equipment. By moving 
the pointer on the screen and selecting a new control Item, the control display unit 
t prompts the user to train the command delivery unit with the equipment's original 
remote control. The control display unit also sends an Identification key value to the 
command delivery unit. At this moment, the user can point the equipment's original 
remote control 800 to the command deliver unit 300 and push a corresponding 
remote control button. An infrared signal is sent to an infrared sensor 361 on the 
- command delivery unit. On F/C. 7a , the infrared signal is converted to electronic code 
by the infrared receiver 360 . This code is stored with the identification key value into 
the memory module 370 . The code stored is retrieved later by the command delivery 
unit to control the equipment. Because the infrared command received by the home 
entertainment equipment received is exactly the same as their original native code, 
any infrared controlled equipment can be controlled by the command delivery unit. 

[0034] 1^ alternative implementation, the control command codes can be stored In the 
display control unit instead of the command delivery unit as shown In FIG. 8a . In this 
case, an infrared receiver 260 and an infrared sensor 261 are added to the unit. 
During the programming procedure shown in FIG. 8b , the home entertainment 
equipment's original remote control 800 \s pointed to the display control unit 200 , 
The infrared signal is received by the infrared sensor 261 and infrared receiver 260 , 
and the command codes are stored in the memory module 270W\X\\ an identification 
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key value. During control session, the MCU retrieves the command code using a key 
value according to the user's selection on the screen. The command code is sent to 
the command delivery unit through the modulator 232 Bind transmitter 222 . The 
command delivery unit in this case simply forwards the command to the target home 
entertainment equipment. 
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